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Abstract 

A  design  is  proposed  for  an  Adaptive  Delegation  Interface  (ADI)  to  support  human 
operators  in  controlling  multiple  UVs.  The  ADI  can  enhance  system  performance  by  (1) 
providing  the  operator  flexibility  to  allocate  UV  automation  in  response  to  changes  in 
workload  and  task  complexity;  and  (2)  adaptively  selecting  information  to  support 
instantaneous  task  needs.  The  proposed  project  will  develop  new  ADI  concepts  and 
evaluate  them  experimentally  in  a  realistic  simulation  environment.  This  paper  describes 
the  concept  of  ADI,  the  current  interface  design,  and  its  integration  within  a  military 
simulation  environment. 

Introduction 

Unmanned  vehicles  and  other  robotic  systems  are  being  introduced  into  Army  systems  to 
extend  manned  capabilities  and  act  as  “force  multipliers”  (Barnes,  Parasuraman  & 
Cosenzo,  in  press;  Cosenzo  et  al,  2006).  Because  of  the  resulting  increase  in  the  cognitive 
workload  demands  on  the  soldier,  increasing  levels  of  robot  autonomy  as  well  as  adaptive 
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automation  aids  may  be  required  to  enhance  soldier  and  system  performance.  The 
resulting  human-robot  team  represents  a  mixed-initiative  system  —  an  intermediate  stage 
between  the  far-off  goal  of  “full”  robot  autonomy  and  the  unacceptable  use  of  completely 
manual  robotic  teleoperation  by  the  soldier. 

Mixed-initiative  systems  can  in  principle  yield  significant  benefits  in  terms  of  mission 
effectiveness.  However,  appropriate  and  sensitive  metrics  are  needed  for  evaluating 
human-robot  team  performance  in  order  to  determine  the  appropriate  levels  of  robot 
autonomy  and  automation  that  will  lead  to  enhance  system  performance.  This  is  because 
mixed  initiative  introduces  a  new  and  unique  aspect  to  the  psychology  of  team 
performance:  the  interaction  of  two  cognitive  systems  —  human  and  autonomous  or  semi- 
autonomous  unmanned  robot.  In  addition  to  the  critical  performance  factors  associated 
with  human  teams  —  which  include  information  exchange,  communication,  supporting 
behavior  and  team  leadership  —  the  mixed  manned/unmanned  team  adds  a  number  of 
challenging  new  dimensions.  Foremost  among  these  is  the  ability  of  the  human  team  to 
predict,  collaborate,  and  develop  trust  with  unmanned  systems  that  may  sometimes 
exhibit  fuzzy  responses  in  unstructured  and  unpredictable  environments. 

Innovative  ADI  Concept 

Miller  and  Parasuraman  (2007)  have  proposed  an  innovative  concept  of  adaptive  human- 
automation  interaction  based  on  the  concept  of  delegation  -  the  concept  is  called 
Adaptive  Delegation  Interfaces,  or  ADI.  Adaptive  delegation  interfaces  are  adaptive 
because  they  are  responsive  to  context  and  user  needs,  and  involve  delegation  in  the  same 
sense  that  a  supervisor  works  with  a  human  subordinate — the  difference  being  that  the 
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human  uses  an  interface  that  allows  for  high-level  communication  with  the  automation  in 
a  common  language. 

The  performance  benefit  of  adaptive  compared  to  static  automation  is  well  documented 
(Parasuraman  et  al.,  2005).  Delegation  interfaces  provide  a  methodology  that  allows 
operators  to  explicitly  task  automation  at  times  of  their  choosing — that  is,  where  the  user 
is  in  charge  of  the  adaptation.  At  present,  however,  there  is  only  a  small  body  of 
empirical  research  on  human  supervision  of  UVs,  particularly  with  automation  support. 

Previous  research  has  pointed  to  the  general  utility  of  delegation  interfaces  in  allowing 
for  effective  supervisory  control  of  multiple  numbers  of  UVs  and  recent  experimental 
work  has  partially  validated  the  concept  for  a  relatively  simple  multi-robot  “capture  the 
flag”  task  (Parasuraman  et  al.,  2005).  These  results  point  to  the  potential  for  delegation 
interfaces  to  enhance  mission  effectiveness  in  several  critical  situations;  for  example:  in 
multi-robot  supervision  by  a  single  operator,  or  where  an  operator  must  time-share 
control  of  a  robot  with  other  essential  operational  tasks,  for  example  while  serving  as  a 
member  of  a  tank  crew. 

However,  the  history  of  human-computer  interaction  design  is  replete  with  instances  of 
interfaces  that  were  initially  thought  to  be  beneficial  but  were  inadequately  tested,  with 
the  result  that  they  were  found  to  be  plagued  with  subsequent  problems  (Schneiderman, 
1993).  It  is  therefore  important  that  the  potential  drawbacks  as  well  as  the  advantages  of 
any  new  interface  concept  are  also  investigated,  so  that  a  balanced  view  can  be  drawn  of 
its  effectiveness  and  usability.  Two  potential  problem  areas  are  increased  or  unbalanced 
mental  workload,  which  can  be  a  feature  of  some  automation  designs  (Parasuraman  & 
Riley,  1997),  and  high  task  switching  times.  Accordingly,  while  the  study  results  are 
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promising,  the  ADI  concept  needs  further  development  as  well  as  validation  in  tactically 
realistic  scenarios  before  it  can  be  applied  in  the  design  of  operational  battlefield  systems. 
This  research  project  is  intended  to  accomplish  those  goals. 

Proposed  Research  Approach 

Previous  work  on  interaction  between  humans  and  automated  agents,  including 
unmanned  vehicles  (UVs),  has  revealed  both  benefits  and  costs  of  automation  for  system 
performance  (Parasuraman  &  Riley,  1997;  Parasuraman  et  al.,  2005;  Sheridan  & 
Parasuraman,  2006).  Automation  is  clearly  essential  for  the  operation  of  many  complex 
human-machine  systems,  including  UVs  and  other  robotic  systems.  But  automation  can 
also  lead  to  novel  problems  such  as  increased  workload  and  training  requirements, 
impaired  situation  awareness  and,  when  particular  events  co-occur  in  combination  with 
poorly  designed  interfaces,  accidents  (Degani,  2004).  Retaining  the  benefits  of 
automation  while  minimizing  its  costs  and  hazards  may  require  the  interface  between 
humans  and  robotic  agents  to  be  adaptive  rather  than  fixed  and  static.  The  performance 
benefit  of  adaptive  compared  to  static  automation  is  well  documented  (Parasuraman  et 
al.,  2005).  However,  if  adaptation  is  executed  without  user  approval  or  knowledge,  the 
cost  of  system  unpredictability  may  outweigh  the  benefit  that  automation  provides.  What 
is  needed  is  a  method  that  allows  operators  to  explicitly  task  automation  at  times  of  their 
choosing — that  is,  where  the  user  is  in  charge  of  the  adaptation.  We  have  recently 
proposed  a  theory  of  adaptive  human-automation  interaction  based  on  the  concept  of 
delegation  (Miller  &  Parasuraman,  2007). 

The  key  idea  behind  delegation  is  that  the  supervisor  has  flexibility  in  the  use  of 
automated  support  in  supervising  multiple  UVs.  The  operator  can  delegate  bigger, 
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coarser-grained  tasks  or  smaller,  more  precise  ones  with  more  or  less  explicit  instruction 
about  their  performance,  depending  on  context  and  task  demands.  Delegation 
architectures  seek  to  provide  highly  flexible  methods  for  the  human  supervisor  to  declare 
goals  and  provide  instructions  and  thereby  choose  how  much  or  how  little  autonomy  to 
give  to  automation,  depending  on  context  and  the  current  situation.  It  is  important  to  note 
that  the  communication  between  human  and  automated  agents,  while  using  a  mutually 
understood  “language”,  are  nevertheless  constrained  by  the  specific  doctrine,  jargon,  and 
tactics  of  the  domain,  and  do  not  involve  completely  unconstrained  communication  as 
between  two  humans. 

Figure  1  shows  the  ADI  concept  in 
a  total  system  context.  The 
Interface  Controller  selects  the 
information  display  and 
automation  mode,  the  Information 
Priority  Selector  determines  what 
information  the  operator  needs  based 
on  its  value  as  calculated  using  the  Task  Decision  Model,  the  Delegation  Modes  Library 
supplies  the  appropriate  modes  of  control  as  determined  by  the  human-robot  Team 
Performance  Assessment  system  and  operator  inputs.  The  ADI  is  adaptive  because  it  is 
responsive  to  immediate  context  and  user  needs,  and  involves  delegation  because  it 
allows  the  operator  with  the  help  of  the  system  to  task  the  robot  at  the  appropriate  level  of 


Figure  1.  ADI  in  Total  System  Context 


automation. 
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ADI  Interface  Design 

We  have  designed  the  interface  of  the  ADI  based  on  common  human  factors  principles 
that  optimize  situation  awareness  and  task  load  for  the  operator  (see  Endsley  et  al.,  2003). 
In  addition,  earlier  interfaces  to  control  robotic  assets  have  inspired  our  design.  Figure  2 
shows  the  current  interface  of  the  ADI. 


Figure  2.  ADI  Interface. 

In  the  following  we  describe  each  component  of  the  interface.  Each  display  is  designed 
with  the  concept  of  flexible  delegation.  The  operator  can  work  within  the  complete 
automation  spectrum  from  maintaining  a  high-level  overview  and  letting  most  of  the 
automation  do  the  work,  to  specifying  plans  and  monitoring  each  of  the  assets 
individually. 


©  Perceptronics  Solutions,  Inc.,  July  2007 


Mission  Planning  &  Execution. 

Any  mission  begins  with  planning  and  therefore  the  operator  can  use  this  panel  to  specify 
his  plan.  We  designed  this  display  so  operators  can  specify  a  plan  at  a  very  level  of  detail, 
or  alternatively,  provide  the  system  with  high  level  parameters  and  let  the  automation 
finish  the  rest  of  the  plan.  The  operator  has  a  choice  between  several  mission  templates, 
e.g.,  reconnaissance,  surveillance,  and  IED  seek  &  destroy.  Each  of  these  templates 
requires  certain  parameters  to  be  specified  by  the  operator  while  others  can  be  handled  by 
either  the  operator  or  the  automation.  When  the  operator  is  finished  with  his  plan,  he  can 
execute.  The  display  then  turns  into  a  monitoring  panel  which  displays  the  high  overview 
status  of  each  robotic  asset. 

System  Monitoring. 

During  a  tactical  mission,  robotic  assets  require  a  degree  of  system  monitoring.  This 
panel  shows  an  overview  of  the  robotic  unit  status.  Should  a  malfunction  occur  with  these 
robotic  assets,  the  operator  has  the  ability  to  drill  down  into  this  display  to  investigate 
occurring  problems.  This  display  also  shows  how  automation  can  be  scaled  for  system 
monitoring.  The  operator  can  choose  to  merely  monitor  the  high-level  overview  of  the 
system  and  let  robotic  assets  handle  any  system  malfunctions  or  investigate  a  specific 
issue  by  drilling  down  into  the  display.  This  panel  then  allows  for  scalable  situation 
awareness  based  on  the  operator’s  needs  and  capabilities  at  a  given  moment  during  a 
mission. 

Intelligence  Gathering  &  Analysis 

Robotic  assets  will  collect  sensor  data  throughout  the  mission  that  can  be  gathered  and 
analyzed  in  this  display.  This  data  can  include  images,  Infra-red  sensor  data,  and  live 
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video  feed.  The  operator  can  again  let  the  automation  analyze  the  data  and  produce  a  list 
of  potential  targets  and  relevant  information,  or  research  the  data  himself  and  produce  a 
list  of  potential  targets  and  information.  This  information  can  then  be  used  to  adjust  the 
plan  for  the  robotic  assets. 

Communications 

Communications  are  important  for  the  operator  in  several  ways.  The  operator  should  be 
able  to  receive  commands  from  a  commander  or  superior  officer.  Updates  on  the  mission 
can  also  be  relayed  by  the  operator  to  superior  officers  through  the  communication 
display.  In  addition,  robotic  assets  can  send  messages  to  the  operator  of  various  types 
including,  but  not  limited  to,  status  messages,  target  confirmation  requests,  and  severe 
system  failures.  Future  implementations  of  the  ADI  may  also  include  the  ability  to 
converse  with  the  robotic  assets  through  this  panel. 

Courteous  Intelligent  Robotic  Agent  (CIRA) 

The  ADI  is  a  comprehensive  display.  There  is  a  risk  of  overloading  the  operator  with  too 
much  information.  We  designed  the  Courteous  Intelligent  Robotic  Agent  to  facilitate 
interaction  with  the  robotic  assets  and  to  attract  attention,  if  needed,  to  critical  issues  in 
the  mission  planning  and  execution,  system  monitoring,  data  gathering  and  analysis,  and 
communications  with  the  robotic  assets.  Three  features  make  the  CIRA  a  handy  tool  for 
an  operator: 

•  The  CIRA  gives  suggestions  on  the  next  course  of  action  for  the  operator 

•  The  CIRA  alerts  the  operator  to  highly  critical  issues  during  a  mission 

•  The  CIRA  shows  the  current  stage  of  the  mission 

•  The  CIRA  can  be  turned  on  or  off  based  on  the  operator  preference 
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Simulation  Environment 

The  ADI  Interface  will  be  integrated  with 
our  Mixed  Initiative  Performance 
Assessment  System  (MITPAS)  (Freedy  et 
al,  2006).  Figure  3  shows  the  command 
and  control  configuration  for  our 
simulation  environment  and  experimental 


UAV 


studies.  The  Battlemaster,  who  plays 

Figure  3.  Command  and  Control  Structure 

the  platoon  leader  (see  below),  will  also  be 

in  charge  of  the  experimental  procedures,  the  progress  of  the  scenario,  and 
communication  with  the  Unmanned  Ground  Vehicle  controller,  who  is  the  actual 
experimental  participant.  In  our  planned  scenario,  the  UGV  Controller  will  be  able  to 
control  several  UGVs  simultaneously  using  the  ADI. 


Figure  4.  MITPAS  UGV  Station  with  ADI. 
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The  command  and  control  stations  used  by  the  Battlemaster  and  the  UGV  controller  are 
very  similar.  This  station  as  shown  in  Figure  4  is  comprised  of  a  tactical  situation  map, 
the  ADI,  and  a  UGV  video  feed  represented  on  three  19  inch  monitors.  The  operator  uses 
a  keyboard,  track-ball  mouse,  and  joystick  to  control  the  system. 

The  middle  screen  shows  the  ADI.  The  operator  can  manage  all  his  tasks  from  this 
display.  The  tactical  situation  map,  generated  by  the  military  simulator  OOS  on  the  right 
monitor,  shows  the  UGVs’  geographic  positions  and  also  gives  an  overview  of  the  entire 
tactical  situation.  On  the  left  monitor,  video  feeds  from  each  UGV  are  available  if  the 
operator  wishes  to  obtain  a  closer  of  the  situation. 

Tactical  Scenario 

Currently  we  have  developed  one  scalable  robotic  team  behavior  —  search  for  IEDs.  We 
have  developed  a  Route  Reconnaissance  and  Obstacle  Clearance  (RROC)  based  on  this 
team  behavior  (see  Figure  5)  that  spans  the  anticipated  range  of  maneuver  control  and 
contingency  handling  events.  The  mission  is  to  ensure  in  an  urban  environment  that  the 
area  is  safe  for  an  approaching  convoy  by  identifying  and  eliminating  dangerous  (IEDs), 
objects  and  surrounding  enemies.  The  available  UV  resources  are  two  TALON  ground 
vehicles  and  a  surveillance  helicopter  working  as  a  team  with  one  human  operator.  In 
brief,  the  robot-human  team  is  part  of  a  reconnaissance  platoon.  Its  mission  to  ensure 
that  a  specific  terrain  area  is  safe  by  detecting  a  mined  car  and  clearing  it  as  well  as 
potentially  eliminating  all  surrounding  enemy  forces.  To  do  this,  the  team  has  to 
coordinate  the  actions  of  the  UAV  and  UGVs,  move  the  UGVs  to  a  checkpoint  where 
they  can  commence  mine  clearing  operations  until  the  mine  is  neutralized  as  well  as 
targeting  and  firing  on  enemy  forces  until  they  are  destroyed. 
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Figure  5.  Scenario  with  Several  Robots  Conducting  a  Reconnaissance  Mission 
The  team  has  to  monitor  and  evaluate  the  maneuvering,  IED  clearing  and  autonomous 
targeting  and  firing  capabilities  of  the  UVs  and  take  over  control  of  the  vehicles  if 
decrements  in  these  autonomous  processes  would  cause  a  mission  delay  or  a  complete 
failure;  e.g.,  firing  competency  can  be  enhanced  by  moving  the  Fire  Cover  UGV  closer  to 
the  enemy.  The  team  task  will  be  specified  as  an  event-based  scenario  containing  planned 
events  which  will  evoke  re-planning  by  the  team  due  to  unpredictable  contingencies:  A 
number  of  contingencies  typical  of  future  robot  systems  will  be  inserted.  These  include: 

•  Obstacle  in  the  road 

•  Communication  loss 

•  Mechanical  failure 

We  plan  to  create  additional  and  more  scalable  robot  team  behaviors  to  support  control  of 
the  human  operator  through  the  ADI. 
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Conclusion 

As  robotic  assets  become  more  autonomous  the  role  of  robotic  operators  will  shift  from 
physically  controlling  one  robot  to  managing  a  robotic  team  with  numerous  assets.  We 
believe  that  the  adaptive  delegation  interface  (ADI)  will  provide  robotic  operators  with 
the  capabilities  needed  to  effectively  manage  several  robotic  assets  simultaneously  in 
dynamic  and  uncertain  environments.  New  interfaces  must  allow  for  flexible,  scalable, 
and  automated  control  of  robotic  assets  to  optimize  human-robot  team  performance.  The 
adaptive  delegation  interface  incorporates  both  adaptive  and  adaptable  automation  to 
address  these  needs.  With  this  interface,  operators  can  scale,  monitor  and  adjust  their 
missions  based  on  preference  and  work  load.  Automation  can  be  activated  or  deactivated 
based  on  mission  events,  mission  history,  and  performance  by  the  operator.  Future 
experiments  will  examine  the  effects  of  the  ADI  on  the  control  of  multiple  robotic  assets 
and  overall  team  performance  and  mission  effectiveness. 
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